Matting and compositing is a well-known problem in both computer graphics and computer vision. Matting separates background and foreground regions in an image or video. Practical methods that yield high quality mattes are extremely important for special effects, television and movie industry. Unfortunately, most prior art methods for extracting high-quality alpha mattes from a video either require user assistance or uniformly colored background screens, e.g., blue or green backgrounds. Mattes are used to composite foregrounds and backgrounds, e.g., a weatherman in front of a weather map.
There are two problems with colored screen matting. First, the foreground cannot include any of the background colors, e.g., the weatherman cannot wear any clothing that is the same color as the background. Second, the background color can ‘spill’ onto the foreground, which considerably changes the appearance of the scene. Even when the background is known exactly, extracting the alpha matte is an under-constrained problem.
Matting has been described extensively since the 1950s, see Vlahos, “Composite photography utilizing sodium vapor illumination,” May, 1958, and U.S. Pat. No. 3,095,304. Blue screen matting was formalized mathematically by Smith and Blinn, “Blue screen matting,” Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques, ACM Press, pp. 259-268, 1996.
Blue screen matting relies on the use of a uniformly colored background and constraining the foreground colors to not be similar to the background color. As shown by Blinn and Smith, imaging a static scene under a known background allows for deriving a correct solution for both the alpha matte and the foreground color without constraints placed on the foreground colors.
Blue screen matting methods have been extended to cover more complicated light transport effects, e.g., refraction, using multiple background patterns, Zongker et al., “Environment matting and compositing,” Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques, ACM Press/Addison-Wesley Publishing Co., pp. 205-214, 1999; and Chuang et al., “Environment matting extensions: towards higher accuracy and real-time capture,” Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques, ACM Press/Addison-Wesley Publishing Co., pp. 121-130, 2000.
Bayesian matting can be used for static scenes, Chuang et al., “A Bayesian approach to digital matting,” Proceedings of IEEE CVPR 2001, vol. 2, pp. 264-271, 2001. That method assumes a user-specified trimap and a low frequency background. Bayesian techniques can also adapt blue screen matting to complex scenes, Chuang et al., “Video matting of complex scenes,” ACM Transactions on Graphics, vol. 21, no. 3, pp. 243-248, July 2002. In that method, a user specifies tri-maps for key frames in the video.
Another method uses a camera array and a stereo process to automatically determine the trimaps, Zitnick et al., “High-quality video view interpolation using a layered representation,” ACM Transactions on Graphics, vol. 23, no. 3, pp. 600-608, 2004.
Another technique, difference matting, also known as background subtraction, solves for the alpha matte and an alpha multiplied foreground given background and trimap images, Qian and Sezan, “Video background replacement without a blue screen,” Proceedings of ICIP, vol. 4, pp. 143-146, 1999.
Difference mattes have limited discrimination at borders between the background and foreground. Another alternative is to use back lighting to extract the matte. Back lighting is a common segmentation method used in many commercial computer vision systems. Back lighting has also been used in image-based rendering systems, Debevec et al., “A lighting reproduction approach to live action compositing,” ACM Transactions on Graphics, vol. 21, no. 3, pp. 547-556, July 2002. That method requires active illumination, which may not always be possible, and can produce incorrect results near object boundaries because some objects are highly reflective at grazing angles of incident light.
Another method is invisible key segmentation, which illuminates the scene with polarized light, or alternatively employs a polarized back-light, and segments the image based on polarization with a chroma-key-like algorithm, Ben-Ezra, “Segmentation with invisible keying signal,” IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 1, pp. 1032-1038, 2000. That method cannot operate under natural illumination, and requires specialized lighting assemblies to produce polarized light.
Therefore, it is desired to provide a matting method that allows arbitrary colors in the foreground of a scene illuminated by ambient, mostly unpolarized light, and that does not introduce the color spill of the uniform background color.